Resource Usage Modeling for Network Monitoring Applications

Building robust network monitoring applications is hard given the unpredictable nature of network traffic and high, ever-increasing data rates. Traffic analysis systems must be designed with load shedding techniques in mind that can reduce the workload of a network monitoring system whilst gracefully degrading the accuracy of the results. We present a novel load shedding approach based on building a model of the resource consumption of monitoring applications and using it to prevent overload. The system extracts a set of features from the traffic in the form of counters, and measures the resource usage of the monitoring tasks. This information is used to build a multiple linear regression based model of the monitoring task. This model can be used to predict the resource usage of the monitoring tasks, and therefore to select the appropriate level of load shedding with great accuracy and in a fine-grained basis. We implement and deploy or system on a high-speed link of a large academic ISP. Our results show that our system predicts resource usage with errors below 5%, and that the predictions can be used to fully prevent uncontrolled packet loss.